{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "SWa-rLfIlTaf"
      },
      "source": [
        "<table align=\"center\">\n",
        "  <td align=\"center\"><a target=\"_blank\" href=\"http://introtodeeplearning.com\">\n",
        "        <img src=\"https://i.ibb.co/Jr88sn2/mit.png\" style=\"padding-bottom:5px;\" />\n",
        "      Visit MIT Deep Learning</a></td>\n",
        "  <td align=\"center\"><a target=\"_blank\" href=\"https://colab.research.google.com/github/MITDeepLearning/introtodeeplearning/blob/master/lab3/Part1_IntroductionCapsa.ipynb\">\n",
        "        <img src=\"https://i.ibb.co/2P3SLwK/colab.png\"  style=\"padding-bottom:5px;\" />Run in Google Colab</a></td>\n",
        "  <td align=\"center\"><a target=\"_blank\" href=\"https://github.com/MITDeepLearning/introtodeeplearning/blob/master/lab3/Part1_IntroductionCapsa.ipynb\">\n",
        "        <img src=\"https://i.ibb.co/xfJbPmL/github.png\"  height=\"70px\" style=\"padding-bottom:5px;\"  />View Source on GitHub</a></td>\n",
        "</table>\n",
        "\n",
        "# Copyright Information"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-LohleBMlahL"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 MIT Introduction to Deep Learning. All Rights Reserved.\n",
        "# \n",
        "# Licensed under the MIT License. You may not use this file except in compliance\n",
        "# with the License. Use and/or modification of this code outside of MIT Introduction\n",
        "# to Deep Learning must reference:\n",
        "#\n",
        "# © MIT Introduction to Deep Learning\n",
        "# http://introtodeeplearning.com\n",
        "#"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ckzz5Hus-hJB"
      },
      "source": [
        "# Laboratory 3: Debiasing, Uncertainty, and Robustness\n",
        "\n",
        "# Part 1: Introduction to Capsa\n",
        "\n",
        "In this lab, we'll explore different ways to make deep learning models more **robust** and **trustworthy**.\n",
        "\n",
        "To achieve this it is critical to be able to identify and diagnose issues of bias and uncertainty in deep learning models, as we explored in the Facial Detection Lab 2. We need benchmarks that uniformly measure how uncertain a given model is, and we need principled ways of measuring bias and uncertainty. To that end, in this lab, we'll utilize [Capsa](https://github.com/themis-ai/capsa), a risk-estimation wrapping library developed by [Themis AI](https://themisai.io/). Capsa supports the estimation of three different types of ***risk***, defined as measures of how robust and trustworthy our model is. These are:\n",
        "1. **Representation bias**: reflects how likely combinations of features are to appear in a given dataset. Often, certain combinations of features are severely under-represented in datasets, which means models learn them less well and can thus lead to unwanted bias.\n",
        "2. **Data uncertainty**: reflects noise in the data, for example when sensors have noisy measurements, classes in datasets have low separations, and generally when very similar inputs lead to drastically different outputs. Also known as *aleatoric* uncertainty. \n",
        "3. **Model uncertainty**: captures the areas of our underlying data distribution that the model has not yet learned or has difficulty learning. Areas of high model uncertainty can be due to out-of-distribution (OOD) samples or data that is harder to learn. Also known as *epistemic* uncertainty."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o02MyoDrnNqP"
      },
      "source": [
        "## CAPSA overview\n",
        "\n",
        "This lab introduces Capsa and its functionalities, to next build automated tools that use Capsa to mitigate the underlying issues of bias and uncertainty.\n",
        "\n",
        "The core idea behind [Capsa](https://themisai.io/capsa/) is that any deep learning model of interest can be ***wrapped*** -- just like wrapping a gift -- to be made ***aware of its own risks***. Risk is captured in representation bias, data uncertainty, and model uncertainty.\n",
        "\n",
        "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2023/lab3/img/capsa_overview.png)\n",
        "\n",
        "This means that Capsa takes the user's original model as input, and modifies it minimally to create a risk-aware variant while preserving the model's underlying structure and training pipeline. Capsa is a one-line addition to any training workflow in TensorFlow. In this part of the lab, we'll apply Capsa's risk estimation methods to a simple regression problem to further explore the notions of bias and uncertainty. \n",
        "\n",
        "Please refer to [Capsa's documentation](https://themisai.io/capsa/) for additional details."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hF0uSqk-nwmA"
      },
      "source": [
        "Let's get started by installing the necessary dependencies:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NdXF4Reyj6yy"
      },
      "outputs": [],
      "source": [
        "# Import Tensorflow 2.0\n",
        "%tensorflow_version 2.x\n",
        "import tensorflow as tf\n",
        "\n",
        "import IPython\n",
        "import functools\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from tqdm import tqdm\n",
        "\n",
        "# Download and import the MIT Introduction to Deep Learning package\n",
        "!pip install mitdeeplearning\n",
        "import mitdeeplearning as mdl\n",
        "\n",
        "# Download and import Capsa\n",
        "!pip install capsa\n",
        "import capsa"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzEcxjKHn8gc"
      },
      "source": [
        "## 1.1 Dataset\n",
        "\n",
        "We will build understanding of bias and uncertainty by training a neural network for a simple 2D regression task: modeling the function $y = x^3$. We will use Capsa to analyze this dataset and the performance of the model. Noise and missing-ness will be injected into the dataset.\n",
        "\n",
        "Let's generate the dataset and visualize it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fH40EhC1j9dH"
      },
      "outputs": [],
      "source": [
        "# Get the data for the cubic function, injected with noise and missing-ness\n",
        "# This is just a toy dataset that we can use to test some of the wrappers on\n",
        "def gen_data(x_min, x_max, n, train=True):\n",
        "  if train: \n",
        "    x = np.random.triangular(x_min, 2, x_max, size=(n, 1))\n",
        "  else: \n",
        "    x = np.linspace(x_min, x_max, n).reshape(n, 1)\n",
        "\n",
        "  sigma = 2*np.exp(-(x+1)**2/1) + 0.2 if train else np.zeros_like(x)\n",
        "  y = x**3/6 + np.random.normal(0, sigma).astype(np.float32)\n",
        "\n",
        "  return x, y\n",
        "\n",
        "# Plot the dataset and visualize the train and test datapoints\n",
        "x_train, y_train = gen_data(-4, 4, 2000, train=True) # train data\n",
        "x_test, y_test = gen_data(-6, 6, 500, train=False) # test data\n",
        "\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.plot(x_test, y_test, c='r', zorder=-1, label='ground truth')\n",
        "plt.scatter(x_train, y_train, s=1.5, label='train data')\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fz3UxT8vuN95"
      },
      "source": [
        "In the plot above, the blue points are the training data, which will be used as inputs to train the neural network model. The red line is the ground truth data, which will be used to evaluate the performance of the model.\n",
        "\n",
        "#### **TODO: Inspecting the 2D regression dataset**\n",
        "\n",
        " Write short (~1 sentence) answers to the questions below to complete the `TODO`s:\n",
        "\n",
        "1. What are your observations about where the train data and test data lie relative to each other?\n",
        "2. What, if any, areas do you expect to have high/low aleatoric (data) uncertainty?\n",
        "3. What, if any, areas do you expect to have high/low epistemic (model) uncertainty?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mXMOYRHnv8tF"
      },
      "source": [
        "## 1.2 Regression on cubic dataset\n",
        "\n",
        "Next we will define a small dense neural network model that can predict `y` given `x`: this is a classical regression task! We will build the model and use the [`model.fit()`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) function to train the model -- normally, without any risk-awareness -- using the train dataset that we visualized above."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7p1XwfZVuB68"
      },
      "outputs": [],
      "source": [
        "### Define and train a dense NN model for the regression task###\n",
        "\n",
        "'''Function to define a small dense NN'''\n",
        "def create_dense_NN():\n",
        "  return tf.keras.Sequential(\n",
        "          [\n",
        "              tf.keras.Input(shape=(1,)),\n",
        "              tf.keras.layers.Dense(32, \"relu\"),\n",
        "              tf.keras.layers.Dense(32, \"relu\"),\n",
        "              tf.keras.layers.Dense(32, \"relu\"),\n",
        "              tf.keras.layers.Dense(1),\n",
        "          ]\n",
        "  )\n",
        "\n",
        "dense_NN = create_dense_NN()\n",
        "\n",
        "# Build the model for regression, defining the loss function and optimizer\n",
        "dense_NN.compile(\n",
        "  optimizer=tf.keras.optimizers.Adam(learning_rate=5e-3),\n",
        "  loss=tf.keras.losses.MeanSquaredError(), # MSE loss for the regression task\n",
        ")\n",
        "\n",
        "# Train the model for 30 epochs using model.fit().\n",
        "loss_history = dense_NN.fit(x_train, y_train, epochs=30)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ovwYBUG3wTDv"
      },
      "source": [
        "Now, we are ready to evaluate our neural network. We use the test data to assess performance on the regression task, and visualize the predicted values against the true values.\n",
        "\n",
        "Given your observation of the data in the previous plot, where do you expect the model to perform well? Let's test the model and see:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fb-EklZywR4D"
      },
      "outputs": [],
      "source": [
        "# Pass the test data through the network and predict the y values\n",
        "y_predicted = dense_NN.predict(x_test)\n",
        "\n",
        "# Visualize the true (x, y) pairs for the test data vs. the predicted values\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.scatter(x_train, y_train, s=1.5, label='train data')\n",
        "plt.plot(x_test, y_test, c='r', zorder=-1, label='ground truth')\n",
        "plt.plot(x_test, y_predicted, c='b', zorder=0, label='predicted')\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Vktjwfu0ReH"
      },
      "source": [
        "\n",
        "#### **TODO: Analyzing the performance of standard regression model**\n",
        "\n",
        "Write short (~1 sentence) answers to the questions below to complete the `TODO`s:\n",
        "\n",
        "1. Where does the model perform well?\n",
        "2. Where does the model perform poorly?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7MzvM48JyZMO"
      },
      "source": [
        "## 1.3 Evaluating bias\n",
        "\n",
        "Now that we've seen what the predictions from this model look like, we will identify and quantify bias and uncertainty in this problem. We first consider bias.\n",
        "\n",
        "Recall that *representation bias* reflects how likely combinations of features are to appear in a given dataset. Capsa calculates how likely combinations of features are by using a histogram estimation approach: the `capsa.HistogramWrapper`. For low-dimensional data, the `capsa.HistogramWrapper` bins the input directly into discrete categories and measures the density. More details of the `HistogramWrapper` and how it can be used are [available here](https://themisai.io/capsa/api_documentation/HistogramWrapper.html).\n",
        "\n",
        "We start by taking our `dense_NN` and wrapping it with the `capsa.HistogramWrapper`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AVv-knsCwOp9"
      },
      "outputs": [],
      "source": [
        "### Wrap the dense network for bias estimation ###\n",
        "\n",
        "standard_dense_NN = create_dense_NN()\n",
        "bias_wrapped_dense_NN = capsa.HistogramWrapper(\n",
        "    standard_dense_NN, # the original model\n",
        "    num_bins=20,\n",
        "    queue_size=2000, # how many samples to track\n",
        "    target_hidden_layer=False # for low-dimensional data (like this dataset), we can estimate biases directly from data\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UFHO7LKcz8uP"
      },
      "source": [
        "Now that we've wrapped the classifier, let's re-train it to update the bias estimates as we train. We can use the exact same training pipeline, using `compile` to build the model and `model.fit()` to train the model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SkyD3rsqy2ff"
      },
      "outputs": [],
      "source": [
        "### Compile and train the wrapped model! ###\n",
        "\n",
        "# Build the model for regression, defining the loss function and optimizer\n",
        "bias_wrapped_dense_NN.compile(\n",
        "  optimizer=tf.keras.optimizers.Adam(learning_rate=2e-3),\n",
        "  loss=tf.keras.losses.MeanSquaredError(), # MSE loss for the regression task\n",
        ")\n",
        "\n",
        "# Train the wrapped model for 30 epochs.\n",
        "loss_history_bias_wrap = bias_wrapped_dense_NN.fit(x_train, y_train, epochs=30)\n",
        "\n",
        "print(\"Done training model with Bias Wrapper!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_6iVeeqq0f_H"
      },
      "source": [
        "We can now use our wrapped model to assess the bias for a given test input. With the wrapping capability, Capsa neatly allows us to output a *bias score* along with the predicted target value. This bias score reflects the density of data surrounding an input point -- the higher the score, the greater the data representation and density. The wrapped, risk-aware model outputs the predicted target and bias score after it is called!\n",
        "\n",
        "Let's see how it is done:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tZ17eCbP0YM4"
      },
      "outputs": [],
      "source": [
        "### Generate and visualize bias scores for data in test set ###\n",
        "\n",
        "# Call the risk-aware model to generate scores\n",
        "predictions, bias = bias_wrapped_dense_NN(x_test)\n",
        "\n",
        "# Visualize the relationship between the input data x and the bias\n",
        "fig, ax = plt.subplots(2, 1, figsize=(8,6))\n",
        "ax[0].plot(x_test, bias, label='bias')\n",
        "ax[0].set_ylabel('Estimated Bias')\n",
        "ax[0].legend()\n",
        "\n",
        "# Let's compare against the ground truth density distribution\n",
        "#   should roughly align with our estimated bias in this toy example\n",
        "ax[1].hist(x_train, 50, label='ground truth')\n",
        "ax[1].set_xlim(-6, 6)\n",
        "ax[1].set_ylabel('True Density')\n",
        "ax[1].legend();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HpDMT_1FERQE"
      },
      "source": [
        "#### **TODO: Evaluating bias with wrapped regression model**\n",
        "\n",
        "Write short (~1 sentence) answers to the questions below to complete the `TODO`s:\n",
        "\n",
        "1. How does the bias score relate to the train/test data density from the first plot?\n",
        "2. What is one limitation of the Histogram approach that simply bins the data based on frequency?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PvS8xR_q27Ec"
      },
      "source": [
        "# 1.4 Estimating data uncertainty\n",
        "\n",
        "Next we turn our attention to uncertainty, first focusing on the uncertainty in the data -- the aleatoric uncertainty.\n",
        "\n",
        "As introduced in Lecture 5 on Robust & Trustworthy Deep Learning, in regression we can estimate aleatoric uncertainty by training the model to predict both a target value and a variance for every input. Because we estimate both a mean and variance for every input, this method is called Mean Variance Estimation (MVE). MVE involves modifying the output layer to predict both the mean and variance, and changing the loss to reflect the prediction likelihood.\n",
        "\n",
        "Capsa automatically implements these changes for us: we can wrap a given model using `capsa.MVEWrapper` to use MVE to estimate aleatoric uncertainty. All we have to do is define the model and the loss function to evaluate its predictions! More details of the `MVEWrapper` and how it can be used are [available here](https://themisai.io/capsa/api_documentation/MVEWrapper.html).\n",
        "\n",
        "Let's take our standard network, wrap it with `capsa.MVEWrapper`, build the wrapped model, and then train it for the regression task. Finally, we evaluate performance of the resulting model by quantifying the aleatoric uncertainty across the data space: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sxmm-2sd3G9u"
      },
      "outputs": [],
      "source": [
        "### Estimating data uncertainty with Capsa wrapping ###\n",
        "\n",
        "standard_dense_NN = create_dense_NN()\n",
        "# Wrap the dense network for aleatoric uncertainty estimation\n",
        "mve_wrapped_NN = capsa.MVEWrapper(standard_dense_NN)\n",
        "\n",
        "# Build the model for regression, defining the loss function and optimizer\n",
        "mve_wrapped_NN.compile(\n",
        "  optimizer=tf.keras.optimizers.Adam(learning_rate=1e-2),\n",
        "  loss=tf.keras.losses.MeanSquaredError(), # MSE loss for the regression task\n",
        ")\n",
        "\n",
        "# Train the wrapped model for 30 epochs.\n",
        "loss_history_mve_wrap = mve_wrapped_NN.fit(x_train, y_train, epochs=30)\n",
        "\n",
        "# Call the uncertainty-aware model to generate outputs for the test data\n",
        "x_test_clipped = np.clip(x_test, x_train.min(), x_train.max())\n",
        "prediction = mve_wrapped_NN(x_test_clipped)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dT2Rx8JCg3NR"
      },
      "outputs": [],
      "source": [
        "# Capsa makes the aleatoric uncertainty an attribute of the prediction!\n",
        "pred = np.array(prediction.y_hat).flatten()\n",
        "unc = np.sqrt(prediction.aleatoric).flatten() # out.aleatoric is the predicted variance\n",
        "\n",
        "# Visualize the aleatoric uncertainty across the data space\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.scatter(x_train, y_train, s=1.5, label='train data')\n",
        "plt.plot(x_test, y_test, c='r', zorder=-1, label='ground truth')\n",
        "plt.fill_between(x_test_clipped.flatten(), pred-2*unc, pred+2*unc, \n",
        "                 color='b', alpha=0.2, label='aleatoric')\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZFeArgRX9U9s"
      },
      "source": [
        "#### **TODO: Estimating aleatoric uncertainty**\n",
        "\n",
        "Write short (~1 sentence) answers to the questions below to complete the `TODO`s:\n",
        "\n",
        "1. For what values of $x$ is the aleatoric uncertainty high or increasing suddenly?\n",
        "2. How does your answer in (1) relate to how the $x$ values are distributed?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6FC5WPRT5lAb"
      },
      "source": [
        "# 1.5 Estimating model uncertainty\n",
        "\n",
        "Finally, we use Capsa for estimating the uncertainty underlying the model predictions -- the epistemic uncertainty. In this example, we'll use ensembles, which essentially copy the model `N` times and average predictions across all runs for a more robust prediction, and also calculate the variance of the `N` runs to estimate the uncertainty.\n",
        "\n",
        "Capsa provides a neat wrapper, `capsa.EnsembleWrapper`, to make an ensemble from an input model. Just like with aleatoric estimation, we can take our standard dense network model, wrap it with `capsa.EnsembleWrapper`, build the wrapped model, and then train it for the regression task. More details of the `EnsembleWrapper` and how it can be used are [available here](https://themisai.io/capsa/api_documentation/EnsembleWrapper.html).\n",
        "\n",
        "Finally, we evaluate the resulting model by quantifying the epistemic uncertainty on the test data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SuRlhq2c5Fob"
      },
      "outputs": [],
      "source": [
        "### Estimating model uncertainty with Capsa wrapping ###\n",
        "\n",
        "standard_dense_NN = create_dense_NN()\n",
        "# Wrap the dense network for epistemic uncertainty estimation with an Ensemble\n",
        "ensemble_NN = capsa.EnsembleWrapper(standard_dense_NN)\n",
        "\n",
        "# Build the model for regression, defining the loss function and optimizer\n",
        "ensemble_NN.compile(\n",
        "  optimizer=tf.keras.optimizers.Adam(learning_rate=3e-3),\n",
        "  loss=tf.keras.losses.MeanSquaredError(), # MSE loss for the regression task\n",
        ")\n",
        "\n",
        "# Train the wrapped model for 30 epochs.\n",
        "loss_history_ensemble = ensemble_NN.fit(x_train, y_train, epochs=30)\n",
        "\n",
        "# Call the uncertainty-aware model to generate outputs for the test data\n",
        "prediction = ensemble_NN(x_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eauNoKDOj_ZT"
      },
      "outputs": [],
      "source": [
        "# Capsa makes the epistemic uncertainty an attribute of the prediction!\n",
        "pred = np.array(prediction.y_hat).flatten()\n",
        "unc = np.array(prediction.epistemic).flatten()\n",
        "\n",
        "# Visualize the aleatoric uncertainty across the data space\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.scatter(x_train, y_train, s=1.5, label='train data')\n",
        "plt.plot(x_test, y_test, c='r', zorder=-1, label='ground truth')\n",
        "plt.fill_between(x_test.flatten(), pred-20*unc, pred+20*unc, color='b', alpha=0.2, label='epistemic')\n",
        "plt.legend()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N4LMn2tLPBdg"
      },
      "source": [
        "#### **TODO: Estimating epistemic uncertainty**\n",
        "\n",
        "Write short (~1 sentence) answers to the questions below to complete the `TODO`s:\n",
        "\n",
        "1. For what values of $x$ is the epistemic uncertainty high or increasing suddenly?\n",
        "2. How does your answer in (1) relate to how the $x$ values are distributed (refer back to original plot)? Think about both the train and test data.\n",
        "3. How could you reduce the epistemic uncertainty in regions where it is high?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CkpvkOL06jRd"
      },
      "source": [
        "# 1.6 Conclusion\n",
        "\n",
        "You've just analyzed the bias, aleatoric uncertainty, and epistemic uncertainty for your first risk-aware model! This is a task that data scientists do constantly to determine methods of improving their models and datasets.\n",
        "\n",
        "In the next part of the lab, you'll continue to build off of these concepts to study them in the context of facial detection systems: not only diagnosing issues of bias and uncertainty, but also developing solutions to *mitigate* these risks.\n",
        "\n",
        "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2023/lab3/img/solutions_toy.png)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nIpfPcpjlsKK"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
